Machine learning in ScalOps, a higher order cloud computing language

Machine learning practitioners are increasingly interested in applying their algorithms to Big Data. Unfortunately, current high-level languages for data analytics in the cloud (e.g., [2, 15, 16, 5]) do not fully cover this domain. One key missing ingredient is means to express iteration over the data (e.g., [19]). Zaharia et al., were the first to answer this call from a systems perspective with Spark [20]. Spark is built on a data abstraction called resilient distributed datasets (RDDs) that reference immutable data collections. The Spark domain-specific language (DSL) defines standard relational algebra transformations—selection, join, group by, etc.—and mechanisms to cache RDDs in memory. The Spark runtime is optimized for in-memory computation and has published speedups of 30× over Hadoop MapReduce for many machine learning and graph algorithms, which is in line with the gains found using MPI or even special case implementations (e.g. [18]).